Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction

We analyze a corpus of referring expressions collected from user interactions with a multimodal travel guide application. The analysis suggests that, in dramatic contrast to normal modes of human-human interaction, the interpretation of referring expressions can be computed with very high accuracy using a model which pairs an impoverished notion of discourse state with a simple set of rules that are insensitive to the type of referring expression used. We attribute this result to the implicit manner in which the interface conveys the system’s beliefs about the operative discourse state, to which users tailor their choice of referring expressions. This result offers new insight into the way computer interfaces can shape a user’s language behavior, insights which can be exploited to bring otherwise difficult interpretation problems into the realm of tractability.